Skip to content

[WIP] Add Datera storage plugin#3410

Closed
jcsdatera wants to merge 7 commits into
apache:masterfrom
jcsdatera:master.datera
Closed

[WIP] Add Datera storage plugin#3410
jcsdatera wants to merge 7 commits into
apache:masterfrom
jcsdatera:master.datera

Conversation

@jcsdatera
Copy link
Copy Markdown

@jcsdatera jcsdatera commented Jun 18, 2019

Features:

  1. Zone-wide and cluster-wide primary storage support
  2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
  3. Rapid storage-native snapshot
  4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
    • Total provisioned capacity
    • Default storage QoS values
    • Replica size ( 1 to 5 )
    • IP pool assignment for iSCSI target
    • Volume Placement ( hybrid, single_flash, all_flash )
  5. Volume snapshot to VM template
  6. Volume to VM template
  7. Volume size increase using service policy
  8. Volume QoS change using service policy
  9. Enabled KVM support
  10. New Datera app_instance name format to include ACS volume name
  11. VM live migration

Description

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

@jcsdatera
[WIP] Add Datera storage plugin
9cc8fa1 
Features:

1. Zone-wide and cluster-wide primary storage support
2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
3. Rapid storage-native snapshot
4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
      - Total provisioned capacity
      - Default storage QoS values
      - Replica size ( 1 to 5 )
      - IP pool assignment for iSCSI target
      - Volume Placement ( hybrid, single_flash, all_flash )
 5. Volume snapshot to VM template
 6. Volume to VM template
 7. Volume size increase using service policy
 8. Volume QoS change using service policy
 9. Enabled KVM support
10. New Datera app_instance name format to include ACS volume name
11. VM live migration
@jcsdatera jcsdatera changed the title [WIP] Add Datera storage plugin Add Datera storage plugin Jun 19, 2019
@swill
Copy link
Copy Markdown
Contributor

swill commented Jun 19, 2019

Hey @rhtyd, can we try to get this PR into the 4.13 branch before the RC is cut?

Let us know if there are any outstanding items needed for this to be included. Thanks...

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 20, 2019

@swill it depends. Is this PR code complete and ready for review.

If this is ready for review we can kick regression smoke tests and merge this on master as soon as this meets our merging guideline (2 lgtms, no regressions).

@yadvr yadvr mentioned this pull request Jun 20, 2019
Closed
4 tasks
@yadvr yadvr added this to the 4.13.0.0 milestone Jun 20, 2019
yadvr
yadvr requested changes Jun 20, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@yadvr yadvr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcsdatera @swill the added plugin is not loaded in client/pom.xml, so the built jar won't make into cloudstack uber/fat jar, please check if this is intended?

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 25, 2019

ping @jcsdatera @swill ?

jcsdatera added 6 commits June 26, 2019 08:30
@jcsdatera
[WIP] Add Datera storage plugin
8bd764e 
Features:

1. Zone-wide and cluster-wide primary storage support
2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
3. Rapid storage-native snapshot
4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
      - Total provisioned capacity
      - Default storage QoS values
      - Replica size ( 1 to 5 )
      - IP pool assignment for iSCSI target
      - Volume Placement ( hybrid, single_flash, all_flash )
 5. Volume snapshot to VM template
 6. Volume to VM template
 7. Volume size increase using service policy
 8. Volume QoS change using service policy
 9. Enabled KVM support
10. New Datera app_instance name format to include ACS volume name
11. VM live migration
@jcsdatera
Add Datera storage plugin
6a37f45 
    Features:

    1. Zone-wide and cluster-wide primary storage support
    2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
    3. Rapid storage-native snapshot
    4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
          - Total provisioned capacity
          - Default storage QoS values
          - Replica size ( 1 to 5 )
          - IP pool assignment for iSCSI target
          - Volume Placement ( hybrid, single_flash, all_flash )
     5. Volume snapshot to VM template
     6. Volume to VM template
     7. Volume size increase using service policy
     8. Volume QoS change using service policy
     9. Enabled KVM support
    10. New Datera app_instance name format to include ACS volume name
    11. VM live migration
@jcsdatera
Merge branch 'master.datera' of https://github.com/jcsdatera/cloudstack…
80838bf 
… into master.datera
@jcsdatera
[WIP] Add Datera storage plugin
e158711 
Features:

1. Zone-wide and cluster-wide primary storage support
2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
3. Rapid storage-native snapshot
4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
      - Total provisioned capacity
      - Default storage QoS values
      - Replica size ( 1 to 5 )
      - IP pool assignment for iSCSI target
      - Volume Placement ( hybrid, single_flash, all_flash )
 5. Volume snapshot to VM template
 6. Volume to VM template
 7. Volume size increase using service policy
 8. Volume QoS change using service policy
 9. Enabled KVM support
10. New Datera app_instance name format to include ACS volume name
11. VM live migration
@jcsdatera
Add Datera storage plugin
4600f4c 
    Features:

    1. Zone-wide and cluster-wide primary storage support
    2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
    3. Rapid storage-native snapshot
    4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of
          - Total provisioned capacity
          - Default storage QoS values
          - Replica size ( 1 to 5 )
          - IP pool assignment for iSCSI target
          - Volume Placement ( hybrid, single_flash, all_flash )
     5. Volume snapshot to VM template
     6. Volume to VM template
     7. Volume size increase using service policy
     8. Volume QoS change using service policy
     9. Enabled KVM support
    10. New Datera app_instance name format to include ACS volume name
    11. VM live migration
@jcsdatera
Merge branch 'master.datera' of https://github.com/jcsdatera/cloudstack…
27e15ea 
… into master.datera
@jcsdatera
Copy link
Copy Markdown
Author

jcsdatera commented Jun 26, 2019

Thank you @rhtyd, I have updated client/pom.xml.

@yadvr yadvr changed the base branch from master to 4.11 June 26, 2019 11:09
@yadvr yadvr changed the base branch from 4.11 to master June 26, 2019 11:09
@yadvr yadvr closed this Jun 26, 2019
@yadvr yadvr reopened this Jun 26, 2019
@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-41

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 26, 2019

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-45

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 26, 2019

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 26, 2019

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-46

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 27, 2019

@blueorangutan test

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 27, 2019

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 28, 2019

@jcsdatera @swill the PR fails to start the management server with the error:

2019-06-27 04:08:36,762 WARN  [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.cloudstack.spring.lifecycle.ConfigDepotLifeCycle#0': Unsatisfied dependency expressed through field 'configDepotAdmin'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
2019-06-27 04:08:36,762 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@646be2c3{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.cloudstack.spring.lifecycle.ConfigDepotLifeCycle#0': Unsatisfied dependency expressed through field 'configDepotAdmin'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:586)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:238)
        at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:709)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:534)
...
Caused by: org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1380)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:670)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:637)
        ... 63 more
Caused by: java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

Can you test and confirm the PR work against master? I'll rekick tests one more time.

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 28, 2019

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jun 28, 2019

@jcsdatera @swill also check Travis, the jobs are failing there with same errors.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jun 28, 2019

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-50

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jul 3, 2019

Ping @jcsdatera @swill - if you'd like this to make into 4.13.0.0 please work with me in getting the build/run errors fixed. Thanks.

@yadvr yadvr changed the title Add Datera storage plugin [WIP] Add Datera storage plugin Jul 3, 2019
@jcsdatera
Copy link
Copy Markdown
Author

jcsdatera commented Jul 5, 2019

Ping @rhtyd @swill please use PR-3470 instead of this one.

Thank you

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jul 5, 2019

Closing this one based on your remark @manojkverma (assuming you're a colleague). cc @swill @jcsdatera

@yadvr yadvr closed this Jul 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yadvr yadvr yadvr requested changes

Assignees

No one assigned

Labels

component:primary-storage status:work-in-progress type:new-feature

Projects

None yet

Milestone

4.13.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@jcsdatera @swill @yadvr @blueorangutan @manojkverma